Weakly Supervised Temporal Adjacent Network for Language Grounding

Temporal language grounding (TLG) is a fundamental and challenging problem for vision understanding. Existing methods mainly focus on fully supervised setting with temporal boundary labels training, which, however, suffers expensive cost of annotation. In this work, we are dedicated to weakly TLG, where multiple description sentences given an untrimmed video without labels. task, it critical learn strong cross-modal semantic alignment between sentence semantics visual content. To end, introduce novel adjacent network (WSTAN) grounding. Specifically, WSTAN learns by exploiting in instance learning (MIL) paradigm, whole paragraph as input. Moreover, integrate complementary branch into the framework, which explicitly refines predictions pseudo supervision from MIL stage. An additional self-discriminating loss devised both branch, aiming enhance discrimination self-supervising. Extensive experiments conducted three widely used benchmark datasets, i.e., ActivityNet-Captions, Charades-STA, DiDeMo, results demonstrate effectiveness our approach.